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ABSTRACT 

Over the last 2 years, 3 Australian universities have 
been cooperating on a Commonwealth Government funded project on 
Distance Teaching and Evaluation Skills to identify and disseminate 
good practice in the professional development of distance educators. 
A specific and supportive professional development intervention to 
foster the evaluation of subjects taught by distance education was 
studied. Experienced distance educators at the University of South 
Australia were asked to comment on components of an evaluation 
questionnaire that sought student responses to a teaching package 
with which they were working. From this, a revised general 
questionnaire and four shorter template versions were developed to 
deal with subject content, the study package and library services, 
the teaching process, and assessment. "Constructing an Evaluation 
Questionnaire" was also prepared. At two other universities, Deakin 
and Monash, professional development activities centered on workshops 
for distance educators to identify concerns and to consider the 
approach undertaken at the University of South Australia. Workshop 
participants concluded that "Constructing an Evaluation 
Questionnaire" and its supporting documents represents an approach 
suitable for inclusion in the general professional development 
resources for the Distance Teaching and Evaluation Skills project. 
(SLD) 
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An approach to the evaluation of subjects taught in the distance mode 

In Australia, the Government has for some time been concerned about the quality of distance 
education provision at higher education level. The position taken here is that an important 
factor in influencing the quality of higher education is the attention given to the improvement 
of teaching. Because the ability to create and manage learning environments for students 
depends on accurate appraisals of those activities in implementation, implicit in any notion of 
good teaching is an obligation on the part of the teacher to evaluate teaching and learning. It 
follows that encouragement of informed reflection on the quality of teaching is an important 
dimension of the professional development responsibilities of distance teaching institutions. 
Over the last two years, three Australian Universities have been cooperating on a 
Commonwealth Government funded project 'Distance Teaching and Evaluation Skills' to 
identify and disseminate good practice in the professional development of distance educators. 
This paper considers that part of the project that focuses on the improvement of distance 
teaching through course subject evaluation. 

There are two elements in this discussion: first, a position taken on the nature of evaluation 
when applied to distance teaching, and second, a description of the staff development' 
activities undertaken in the three Universities to encourage such evaluation activity. 

Evaluation of distance teaching 

Remarkably little has been written about the evaluation of distance teaching, possibly 
because the concept of teaching at a distance is problematic. There are essentially two 
schools of thought: that which considers teaching to have an attenuated role in distance 
education because of the preplanned nature of the process and the determination of teaching 
events with, at best, students in mind, and alternatively, that which argues that the hallmarks 
of distance education, its responsiveness and adaptability to students at hand, its concern for 
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them as individuals, and the individualisation of structured learning processes constitutes an 
emphasis on teaching as central to distance education practice. 

The first school of thought would maintain that evaluation of distance education is essentially 
about the evaluation of learning and any management processes intended to promote 
learning and that centre upon the conditions and outcomes of study. The second school 
would seek to make appraisals about the particular interventions and interactions which 
define teaching at a distance and are often carried out through the use of various 
technologies. 

Similarly, it has to be recognised that evaluative activity can take many forms. As one of us 
(Nunan, 1992:1) has written: 

Evaluation is deciding the worth of something. Evaluation activities are characterised 
by the purposes of the evaluation, the standards that are applied in making 
judgements and whose interests might be served by using such criteria, and the 
degree to which the information collected and the judgements made are available to 
various stakeholders. There are related technical issues of where, when and how 
data is obtained and who collects it. Hence evaluation activity can be carried out at 
different levels and for different purposes. 

This complexity of evaluative activity coupled with the problematic nature of teaching at a 
distance is compounded by the taken-for-granted standards which usually define good 
teaching. In higher education, which is our area of interest, most of the literature on 
evaluation of teaching focuses upon face to face teaching. This focus has led to the 
identification of factors which students use to describe good practice and often there is the- 
assumption that such factors can also be applied to teaching at a distance. While it is our 
position that there are no special characteristics that logically lead to conceptions of teaching 
or learning which differ from those we would use generally, nevertheless we believe that 
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assumptions about good teaching derived from face to face teaching may not transfer 
completely to teaching at a distance and there is a need to be sensitive to differences in 
practices which require special considerations. 

Moses (1984:29) summarises factors commonly used to describe superior university teaching 

in the following equation: 

Effective teaching = competence in subject matter 
+ communication skills 
+ commitment to facilitating student learning 
+ concern for individual students 

These factors are useful in understanding face-to-face teaching and have parallels in distance 
education, although configured differently there. Anecdotal evidence suggests that those who 
evaluate distance teaching and learning at university level commonly focus upon: 

1 . evaluating the learning packages supplied to students, 

2. evaluating the support mechanisms employed by the institution and tutor to 
facilitate learning, and 

3. evaluating the teaching and assessment arrangements experienced by 
students through their assignment load. 

Our interpretation would be that learning packages reveal lecturer competence in subject 
matter, the application of communication skills to support learning, and commitment to 
flexibility in addressing different learning needs. The support mechanisms adopted by both 
institutions and tutor reveal the extent to which special needs arising from the learning milieux 
of individual students are acknowledged. The focus on assignment expectations is 
fundamental in that this is the irreducible minimum experience that occurs as a formal 
requirement of participation in the study program and in responding to assignments teachers 
may reveal competencies in any or all of the factors that Moses descr ibes. 
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Staff development activities in evaluation of distance teaching and learning 

In the "Distance Teaching and Evaluation' Project, we focussed on how best to foster the 
evaluation of teaching and learning at a distance which, as indicated in the first paragraph of 
this paper, was seen to be the responsibility of the teaching academic. The strategy we 
adopted was to develop a specific and supportive professional development intervention 
which was trialed at the University of South Australia. This involved an iterative process 
directed towards the clarification of the characteristics used to define good distance teaching 
and learning on which student feedback might be sought. The context for this was a particular 
evaluation strategy already familiar to the academics involved and from which specific and 
detailed responses from participants were obtained, assessed and acted upon. 

Our perceptions of this intervention were tested at Monash and Deakin Universities, where 
academics who taught at a distance were guided a through a process that encouraged their 
critical reflection on the improvement of distance teaching. Our intention was to avoid a 
concentrated focus on a specific evaluation strategy by making the question of what and how 
to evaluate distance teaching and learning problematic. What was sought were more general 
responses to issues arising from the improvement of teaching and learning at a distance and 
where responses moved to specific characteristics of teaching and learning we were able to 
compare these with those discussed at the University of South Australia. Both approaches 
are elaborated below. 

The intervention developed at the Distance Education Centre of the University of South 
Australia was based on the following assumptions: 

1 . good distance teaching involves a range of evaluative activities which should 

be initiated and controlled by teachers, 
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2. 



3. 



4. 



support mechanisms for fostering evaluative activity should not diminish 
teacher responsibilities for the design of evaluative strategies and 
instruments and different approaches should be supported, 
any support mechanisms developed should undergo periodic review involving 
those who use them, and 

there should be encouragement provided to move beyond the techniques or 
processes for which support arrangements exist. 



The details of the intervention were as follows. 

A group of experienced distance teaching academics was asked to comment on the suitability 
of components of an evaluation questionnaire that sought student responses to any distance 
teaching package with which they might currently be working. The questionnaire had been 
developed by a senior academic working with the Distance Education Centre, modified in 
trials, and subsequently used by dozens of lecturers to review their teaching over three years. 
The questionnaire was in the form of a glossary on the University's materials production 
system which could be incorporated at an academic's discretion into a teaching package, ' 
either in its base form, or with changed stimulus questions determined by the lecturer - 
concerned. In essence, the academic could change the text of standard questions, omit 
questions, or replace them with others but retain the overall format of the questionnaire. The 
reference group was specifically charged to consider whether the areas about which 
information was sought were appropriate and how individual questions within them might be 
refined to yield more helpful information. 

The group had been provided with copies of the questionnaire in advance and met together to 
discuss their responses to it. Their concerns were principally about four matters: 
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1 the generality of the questionnaire and their need for specific information about 
particular aspects of their teaching, 
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2. the common and repetitive use of the questionnaire and the impact of this on 
students, 

3. the need for simplicity and ease in varying the questionnaire to their own 
purposes, and 

4. the specific areas around which information might reasonably be sought. 

In reaction to this, staff of the Distance Education Centre produced five evaluation templates 
which incorporated a revised general and four shorter instruments, each of which dealt with 
one aspect covered in the larger questionnaire. Sections covered in that document were (1) 
subject content, (2) the study package and library services, (3) the leaching process, and (4) 
assessment. There were optional questions in each section and opportunity for students to 
make general comments. A final section dealt with sen/ices provided by the Distance 
Education Centre and was not intended for use by the academic staff member. 

The separate instruments were individually smaller but allowed for more specific feedback on 
the different topics covered, which were presented as (1) subject matter, (2) teaching 
arrangements and interactions, (3) resources for learning, and (4) assignments and 
assessment. 

The five templates were produced in a document Constructing an evaluation questionnaire 
(Distance Education Centre, 1992) which invited academics to consider (1) what matters they 
desired feedback from students upon, (2) whether they sought general feedback or wanted 
more detailed information, and (3) whether they wished to use the templates in their present 
fashion or amend them to pursue matters relevant to their own program. A final option 
allowed for a staff member to develop a quite different set of questions and advise the 
Distance Education Centre on a preferred format so a unique instrument could be prepared. 

This document was circulated to the original group of participating academics, whose 
comments led to some revision and then its distribution within the University. The approach 
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described above was devised by Ted Nunan and achieved strong support as a potentially 
most helpful initiative from the academic reference group. It reflected the assumptions listed 
earlier but was also a deliberate professional development intervention by those with 
academic leadership responsibilities in the Distance Education Centre. In particular, it allowed 
individual academics to test student responses to aspects of their teaching that had been 
established by the reference group as general concerns but with the capacity to vary 
questions about those areas to suit their own subject and teaching approaches. 

The professional development activities undertaken at Deakin and Monash Universities were 
deliberately different in nature and intention. As indicated above, the matter of the 
improvement of distance teaching and learning was made problematic. Discussion occurred 
without reference to some particular approach to evaluation and because participants brought 
very different experience and expertise to the workshops both the processes employed by the 
organisers and the responses obtained at the two universities varied. 

Two sessions were conducted at each University by Ted Nunan, Helen Lentell and Jane 
Keedle with purposes agreed by the three as: 

1 . encouraging participants to undertake critical analysis of practice to reveal 



issues about evaluation, 



2. 



encouraging participants to address evaluation as something they would do 



and own as an integral part of improving their teaching at a distance, 



3. 



encouraging participants to identify areas on which information from students 



would be helpful in taking action to improve their teaching practice and 



clarifying the kinds of questions which might most readily elicit such 



information, and 
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providing activities suitable for staff development by heads of schools or 



faculties or be useful as a source of information for staff engaged in 



improving their evaluation activities. 
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At both universities the sessions encouraged academics to analyse critically an example of 
the practice of one of their group and consider ways in which this and other features of 
distance education might be usefully evaluated by themselves as teachers. 



Among the responses were: 



1 . a marked hesitancy to 'own* evaluation issues which might influence or change 
personal distance teaching practice, shaped by a view of evaluation as a 
specialist, difficult, and time-consuming activity and a lack of conceptual clarity 
about the nature of distance teaching, and 

2. a concern with evaluation issues in general, rather than in relation to distance 
teaching and learning, including the impact of institutional support and culture 
upon evaluation, the need to consider a range of stakeholders, issues 
surrounding the collection and use of information, and criteria for interpretation of 
information. 



These issues are important and reflect the complexity of evaluative activity. It was the task of 
the team to assist the group in grounding and limiting these concerns by relating themlo the 
evaluation of factors which influenced the quality of distance teaching and learning. 



The groups identified such areas as the following: quality of feedback to students, improving 
assessment as a factor in enhancing distance teaching, and the role of student expectations 
in framing their approach to learning. When encouraged to move to specific matters on which 
they might seek data as part of their evaluative activity in relation to these areas, the 
academics involved suggested topics closely related to those identified in South Australia and 
included in the templates. The document, Constructing an evaluation questionnaire , was 
distributed to participants who indicated that it addressed the concerns they had identified 
and could be used in gathering information useful in improving their own distance teaching 
practice 
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The discuosions generated much more than this, of course. For example, in relation to the 
evaluation of teaching and learning, groups saw a need for the following: sta^ awareness of 
the options available, the existence of support for evaluation activities, agreement on the 
criteria for successful learning and the importance of taking account of long-term effects of 
teaching and learning, the ability to interpret any data and its meaning in terms of what to 
change in teaching or courses, the capacity to evaluate processes as well as products, 
together with a range of practical concerns, eg. achieving adequate response rates. 

The consequence of the Deakin and Monash University workshops was that the project group 
considered the approach undertaken at the University of South Australia had legitimacy as a 
supportive strategy in assisting academics who sought to use evaluation techniques in an 
endeavour to improve their distance teaching. The document, Constructing an evaluation 
q uestionnaire, was considered appropriate for inclusion in the general professional 
development resources to be made available under the auspices of the Distance Teaching 
and Evaluation Skills project. 
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